Nonlinear low-dimensional regression using auxiliary coordinates

When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, sliced inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from sliced inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches. We consider the problem of low-dimensional regression, where we want to estimate a mapping between inputs x ∈ Rx and outputs y ∈ Ry that are both continuous and high-dimensional (such as images, or control commands for a robot), but going through a lowdimensional, or latent, space z ∈ Rz : y = g(F(x)), where z = F(x), y = g(z) and Dz < Dx, Dy. In some situations, this can be preferable to a direct (full-dimensional) regression y = G(x), for example if, in addition to the regression, we are interested in obtaining a low-dimensional representation of x for its own sake Appearing in Proceedings of the 15 International Conference on Artificial Intelligence and Statistics (AISTATS) 2012, La Palma, Canary Islands. Volume XX of JMLR: W&CP XX. Copyright 2012 by the authors. (e.g. visualization or feature extraction). Even when the true mapping G is not low-dimensional, using a direct regression requires many parameters (DxDy in linear regression) and their estimation may be unreliable with small sample sizes. Using a low-dimensional composite mapping g ◦ F with fewer parameters can be seen as a form of regularization and lead to better generalization with test data. Finally, a common practical approach is to reduce the dimension of x independently of y, say with principal component analysis (PCA), and then solve the regression. However, the latent coordinates z obtained in this way do not necessarily preserve the information that is needed to predict y. This is the same reason why one would use linear discriminant analysis rather than PCA to preserve class information. We want low-dimensional coordinates z that eliminate information in the input x that is not useful to predict the output y, in particular to reduce noise. In this sense, the problem can be seen as supervised dimensionality reduction. Consider then the problem of least-squares regression (although our arguments should apply to other loss functions). The simplest approach to estimate the dimensionality reduction mapping F and the regression mapping g is to minimize the objective function